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Abstract 

The potential benefits that telecommunication services bring to rural and remote communities are 
generally perceived as beneficial though difficult to ascertain. This article reviews the rationale and elements 
for measuring the impact of telecommunication services on rural and remote communities, and proposes 
some concrete ways forward based on current practice. 
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One does not build bridges by counting the number of people who swim across the river 
(anonymous, cited in Sawhney, 2001). 

...communication should be measured by the successful coordination of efforts (Peters, 1999). 
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1. Introduction and context 

The potential benefits that telecommunication services bring to rural and remote communities 
are generally perceived as beneficial though their actual impact is difficult to ascertain. 
Demonstrating causality is a challenging proposition in that many variables affect the relationship 
between technological investments and the subsequent changes that they contribute to in a 
community. This paper explores efforts at measuring impact with reference to an ongoing project 
in remote communities in Canada, where advanced telecommunication services are being 
introduced. The authors have been involved as partners with a project that seeks to demonstrate 
how information and communication technologies and services can improve the livelihoods of 
aboriginal groups in remote communities. 

K-Net is an aboriginal network that is providing broadband connectivity to First Nations 
communities in the remote regions of northwestern Ontario, Canada. The network is formally 
known as the Kuh-ke-nah Network of SMART First Nations. Kuh-ke-nah is an Oji-Cree 
expression for “everybody” and that is the goal of this network—it is for everybody (Ramirez, et 
al., 2003). 

The communities are part of Nishnawbe-Aski Nation (NAN), located in northern Ontario, 
across an area roughly the size of France. NAN includes a population of approximately 25,000. 
The majority of this population is aboriginal and lives in remote communities with 300-900 
inhabitants. For many communities, the only year-round access into or out of their area is by 
small airplane, though most have winter road access for 5 or 6 weeks during the winter season. 


2. Why is measuring important? A rationale 

The setting is rural and remote communities that are characterized by low population densities 
and large geographic areas. In these places, the market often fails to respond to a weak or 
unproven demand for telecommunication services, particularly when policy and regulatory 
contexts do not highlight universal access. The telecommunication services refer to basic 
telephones or computers with Internet access in emerging market countries, broadband Internet 
access in developed countries, and computer-based applications of telecommunication services. 

In the case of the expansion of telecommunication services to rural and remote areas the market 
generally responds with the minimum investments required to meet local demand or comply with 
regulators’ requirements. A case in point is the 1999 ruling for High Cost Serving Areas by the 
Canadian Radio-television Telecommunications Commission (CRTC) that defined the “basic 
telephone service” that private carriers needed to comply with for the next 3 years, namely as 
single line touch-tone service with local access to the Internet (CRTC, 1999). This is the standard 
of service that any subscriber in Canada can expect to receive by 2003. In other words, a rural 
subscriber gets access to dialup when those living in most urban and small cities are already 
enjoying broadband services. This is indicative of a worldwide trend where rural and remote 
telecommunication infrastructure and services lag behind urban ones. It is also the reason why 
investments in telecommunication services in these areas tend to involve some sort of donor or 
governmental incentive programme. Such programmes are usually characterized as “universal 
access” programmes. 
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The provision of “universal access” to telecommunication services is an important objective of 
communication and economic development policy and legislation in many developed countries 
and emerging market countries. Even now in Africa, it is common for there to be fewer than one 
telephone per 100 people, compared to 10 lines per 100 people in Latin America and over 64 in the 
United States. There is a well-documented correlation between telecommunication access and 
national economic development. 

Having tangible evidence about the benefits of telecommunication services appears to offer 
promise in several ways. As a planning tool, it can inform decisions to build infrastructure. As an 
evaluation tool, it can corroborate the benefits of an investment. Finally, as a process 
documentation tool it can provide lessons for improved replication of pilot experiences. In this 
paper, the authors show that these different benefits are of different relevance to customers in 
rural and remote communities and that there is merit in addressing measurement in a strategic and 
select manner. In other words, measurement is important in that it provides an information input 
to support three functions: planning, monitoring and evaluation, and learning (National Research 
Council, 1998). 

There is another very important dimension to measurement that is best summarized in terms of 
“what you measure is what you get.' Too often, rural telecommunication programmes measure 
only technical dimensions such as teledensity. Programmes and policies focused on increasing 
teledensity have demonstrated a poor track record in responding to the economic and social 
development priorities of rural communities. Teledensity measures can lead toward “line¬ 
dumping” where operators and governments seek to meet teledensity targets regardless of the 
needs of specific locations, the commercial viability of service in those locations, and the needs of 
users in those locations. The notion of effective use has begun to emerge as an additional metric 
that goes beyond teledensity to address the extent to which people have access to the 
infrastructure and can put it to work in practical ways (Gurstein, 2003). 

If the goal of rural telecommunication policies and programmes is, in fact, to improve social 
and economic development, then there is a need to consider and review processes for selecting 
goals, objectives and measurement targets. 

The rationale for government involvement in rural telecommunication access responds to the 
perception that telecommunication services have a positive role to play in community and 
regional development. When private investors do not fulfill that role, others have to intervene and 
treat it—at least temporarily—as a public good. As a result, a number of arrangements and 
partnerships between private organizations and public agencies have evolved: they range from 
competitive tenders where the bidder with the lowest subsidy requirement wins the license to build 
and/or operate a rural phone system (the case of rural telephony in Eastern Nepal, Peru, Chile, 
and Uganda), all the way to direct investment and ownership by state agencies of hbre optic 
networks—as is the case of the Iowa Communications Network, USA (Sawhney, 2001). 

Beyond capital investment, there are examples of successful partnerships that cover the 
operating costs of installed infrastructure, applications and services. This applies both at the level 
of public co-hnancing of local telecentres (Proenza, 2001) and at the level of fibre-optic networks 
where government agencies take on the role of anchor tenants (Sawhney, 2001), thus insuring that 
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those same networks are available to the private sector, non-governmental organizations and 
consumers. 

There is evidence that the community and rural development potential of telecommunication 
services depends on much more than investments in infrastructure. Most often, human resource 
development and awareness building are essential ingredients. The digital divide is not only about 
physical access to phone lines and computers, it is also about becoming aware of the potential 
values of telecommunication services, having access to training on a range of computer and 
information-related skills, and to having an occupation where the relevant uses for the technology 
yield added value or savings (van Dijk, 1999; van Dijk, 2001; Gomez & Martinez, 2001). 

Even basic awareness building and training on the operation of telephone sets can be critically 
important in rural areas of developing countries where many people have never used a telephone. 
In Uganda, MTN Publicom, a cellular-based payphone business, provides rural “caravans” 
combining music and entertainment with basic instruction on how phones operate and how they 
can be used to enhance family life and businesses. 

The above introduction points at several significant dimensions that require attention: 
availability and access to the infrastructure, the services and applications, efforts to build 
awareness of the value of specific services, and the training required to make them fully accessible 
and relevant. Each of these dimensions involves partnerships among stakeholders, and whenever 
there are partnerships and collaborative arrangements involving public money, it becomes 
necessary to track and document the performance of the investment. 


3. Who benefits from measurement and when? 

Several stakeholder groups stand to benefit from information on the potential and actual 
benefits of telecommunication services: private investors, government agencies and donors, service 
providers (health, education, local government, libraries, economic development, NGOs), and the 
users themselves (Table 1). 


Table 1 

The benefits of having measurement or estimates for different decisions and stakeholders 


Decisions stakeholders 

Planning 

Evaluating 

Replicating 

Private investors and 

To project a return on an 

Verification of the return 

Increasing the return, often 

venture capitalists 

investment and secure 
financing 

on an investment 

based on consumer surplus 
(willingness to pay) 

Government/donor 

Justifying decisions to fund 
requests 

Defending decisions 

Describing benefits beyond 
the economic return 

Services providers (the 
supply side) 

Justifying requests for 
funding 

Demonstrating impact 
(quality of service, cost 
savings) 

Projecting future benefits 

Users (the demand side) 

Justifying request for 
funding 

Priming the pump 

Appreciating value 

Seeking more options 
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Since the decisions and the stakeholders involved differ, the type of measurement or estimates 
will need to be specihc to their needs. At the same time, they need to be sufficiently relevant and 
consistent to support the agreements among them. In this paper the authors argue that 
stakeholder involvement is necessary to determine goals and relevant indicators, as a basis for 
measuring impact. Furthermore, it is argued that existing conceptual frameworks and indicator 
categorization tend to ignore the specific needs and conditions of rural and remote communities. 

3.1. Conceptual frameworks and indicator categories 

From the methodological side researchers are working with the legacy of methods that measure 
the impact of rural telephony. Some of the methods aim to estimate the rate of return of 
investments in rural telephony (INTELECON, 1995; Kayani & Dymond, 1997), while others rely 
on national level statistics to demonstrate a positive correlation between infrastructure 
investments and economic indicators (Cronin, McGovern, Miller, & Parker, 1995; Parker & 
Hudson, 1995; Dholakia & Harlam, 1994; Hardy, 1980; Gilling, 1975; Saunders, Warford, & 
Wellenius, 1994; Wellenius & others, 1993). Those methods and correlations are proven; yet while 
access to telephony remains a key economic and infrastructure dimension telecommunication 
services, their impact is far more complex. In fact, from the conceptual side, researchers and 
practitioners have yet to fully grasp the fact that telecommunication services contribute to 
multiple social, economic, and livelihood dimensions that are interrelated in complicated ways 
(Hillier, 2000). The acknowledgement that multiple variables intervene is reported even in the 
narrower field of adoption of information systems (Eegris, Ingham, & Collerette, 2003). 
Moreover, the research literature tends to focus on urban dimensions or on national level 
statistics; there are relatively few publications that focus specifically on the rural development 
dimension (Richardson & Paisley, 1998; Ramirez, 2001), though the number is now growing 
(Euentes-Bautista, 2001; Ramirez, 2003).^ 

In short, what is being measured is a wide range of indicators that are chosen by the specific 
interest of each party as they are perceived to be of relevance to the subject matter. Policy makers 
may be satisfied with standard indicators of teledensity to satisfy universal access policies; 
commercial carriers will need to know the rate of return on investment; residential users will be 
mainly interested in the price and quality of service; while agencies involved in health and 
education may need to document the cost savings brought about by the use of the technology. 
There are multiple players and multiple targets, and measurement needs to embrace this 
complexity. 

There are several frameworks in the literature that organize the multiple dimensions and issues 
that are affected by telecommunication services. Some of the frameworks attempt to organize 
indicators into coherent categories. 

Fig. 1 illustrates the major dimensions and services along four quadrants: service and product 
on one axis, and transport and content on the other. This is an important first step that depicts the 
different dimensions that any one product or service belongs to. However, it remains a conceptual 
“locator” of variables. The same authors move from this conceptual framework to mapping 
indicators using a “footprint” diagram (Fig. 2). This kind of diagram serves best as a means of 

^See also: http://www.telecommons.com/reports.cfm?itemid = 213. 
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Fig. 1. The multiple dimensions of ICTs (Mansell & Wehn, 1998). 


visualizing the relative country scores and drawing comparisons between them. Fig. 3 depicts an 
Australian effort that uses the same axes presented in Fig. 1 and organizes variables that are 
tracked by Australian government. Houghton (1999) developed this arrangement on the basis of 
logical relationships among the 16 groups of variables. He found, however, that the adoption of 
this tool within the government system was short lived (pers. comm.). Fig. 4 provides a summary 
of the groupings presented by the e-readiness guide of Harvard University (www.readinessgui- 
de.org). This is a web-based instrument that guides users through a range of variables, grouped 
under five major categories. The user learns about the relative e-readiness status of a given 
country. 

The last three examples share the following features: they capture national level indicators, they 
provide a “status” report, they acknowledge multiple variables, and they do not capture a 
systemic analysis (though Houghton’s includes a review of relationships among variables). None 
of them, however, focus specifically on rural and remote contexts, nor do they capture the 
complex interrelationships among the different variables. A summary of the relative contribution 
of each one of these methods is provided in Table 2. 


4. A conceptual framework that captures multiple, interrelated dimensions 

The debate surrounding the digital divide has invited a critical review of the dimensions that 
require attention, what Gomez and Martinez (2001) refer to as the social vision of the Internet. 
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“Ideal" 

knowledge 

indicator 



Fig. 2. Mapping indicators: a footprint analysis INEXSK INfrastructure, Experience, Skills, Knowledge (Mansell & 
Wehn, 1998: p. 37). 


They highlight several themes that require attention: going beyond connectivity and addressing 
the training needs and organizational behaviours that enable communities to harness the 
technology, promoting enabling environments such as expanding infrastructure to rural areas 
through universal access policies, and minimizing threats and risks to ensure human and 
organizational investments are sustainable. A sobering perspective is appearing: 

...the approach to ICTs must be information centred, integral to its environment, integrated 
with development objectives, intermediated, inteconnected, and indigenised (Heeks, 2002: 

p. 11). 

...[information and communication technologies] ICTs have many revolutionary implications, 
but in order to achieve their full potential benefits it is necessary to focus on user-oriented and 
cost-effective applications rather than on technology-driven applications (Mansell & Wehn, 
1998: p. 95). 

How to capture these requirements? There is a need for a new approach that acknowledges 
pluralist perspectives and the complexity of the many dimensions that are involved; in other 
words, a new epistemology (Ramirez, 2003). The authors’ experience with community-based 
networks in rural and remote parts of Canada has led to an emphasis on four dimensions: the 
technology, the policy context, the community context, and most importantly the nature of the 
organizations that ‘mediate’ among these variables. This approach places emphasis on the 
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Fig. 3. The Australian technology map (Houghton, 1999: p. 692). 


relationship among these dimensions and the evolution of the network (Fig. 5). This framework 
describes the dynamics behind the evolution of community-based networks. These dynamics refer 
to social and planning dimensions that have a foundation in community development. 

The framework in Fig. 5 was developed on the basis of applied research that emphasizes the 
following interrelated dimensions: Community involvement in voicing needs through local “spaces 
and places” where different organizations can come together, and where they can vision their 
goals and develop monitoring and evaluation (M&E) tools to determine that they are reaching 
those goals. The Policy environment, both as a system of rules that influences the other 
dimensions, and as a continuum of programmes that become building blocks for organizational 
development at the rural community level. Technology and Infrastructure, namely, the 
applications that respond to organizational needs, and how these drive the need for hardware 
and bandwidth. Lastly, and at the centre: organizational issues, namely, how community networks 
emerge and provide a number of services, how they change through time, how they learn, and how 
coherent they are. The community networks that have been studied are best described as 
“mediating organizations” in that they mediate between the needs of the community, the policy 
environment, and the technology. This framework depicts a dynamic system that evolves, and in 
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Information 

Internet 

Internet 

Network 

Hardware 

Service and 

Infrastructure 

availability 

affordability 

speed 

and software 

support 


INFRASTRUCTURE 


NETWORK LEARNING 


Schools access to ICTs 


Enhancing education with 
ICTs 


Developing the ICT 
workforce 


NETWORK SOCIETY 


People and 
organizations online 

Locally relevant 
content 

ICTs In everyday life 

ICTs In the 
workplace 

NETWORK ECONOMY 

ICT employment 
opportunities 

B2C electronic 

commerce 

B2B electronic 

commerce 

E-government 


NETWORK POLICY 


Telecom regulation 


Telecom trade policy 


Fig. 4. The grouping of indicators in Harvard’s E-readiness guide. Adapted from: Information Teehnologies Group 
(n.d.). 

the authors’ experience the most successful mediating organizations are the ones that evolve and 
learn fastest (Ramirez, 2001; Ramirez & Richardson, 2000). 

The above conceptual framework provides a platform for measuring impact by drawing 
attention to four themes. 

1. Community involvement in indicator selection'. This is one area where telecommunication 
services research is embryonic. In other fields of study there are examples where stakeholders have 
negotiated the indicators and who tracks them (Besleme, Maser, & Silverstein, 1999; Guijt & 
Sidersky, 1996). The emphasis on the concerted action among stakeholders as a basis on which to 
gauge impact is beginning to be documented in the telecommunications literature (Taghioff, 2001; 
Chow, Ellis, & Walker, 2000; Durance & Pettigrew, 2001). In Canada the term “community 
engagement” is gaining importance as a fundamental step of developing “Smart” communities [a 
term that refers to communities that use information and communication technology] (http:// 
smartcommunities. ic. gc .ca) . 

2. Organizational dimensions are also gaining attention; evaluating organizational capacities is 
an important area of research in other fields (Horton, 2001). Oftentimes, the capacity of the 
organizations to evolve and adapt to change has been highlighted as a critical element to capture 
in rural and remote ICT research (Ramirez, 2001; Moore, Ramirez, Coghlan, Oliphant, & 
Whiteford, 2001; Ramirez, Murray, Kora, & Richardson, 2000; Young, et ah, 1997). 

3. Individual skills and capacities are an important dimension to capture as they are central 
elements for accessibility (van Dijk, 2001) and this is an area where evaluation tools from adult 
education and community development have proven to be relevant (Ramirez et al., 2000). 

4. Predictability. Information and communication technology has historically led to unexpected 
outcomes in terms of new telecommunications services. Bar et al. (2000) highlight this fact in their 
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Table 2 

The relative eontribution of existing methods to assess ICT uses and roles 


No. 

Title 

Source 

Contribution 

Remark 

1 

Dimensions of 
technology practice and 
experience 

Pacey (1999) 

Expresses the multi-dimensional 
nature of technology 

Multi-dimensionality 

2 

Creating the networks 
for an information 
society 

Melody (1996) 

Shows the infrastructure 
requirements before applications 
and services are possible 

Infrastructure vs. 
applications and 
services 

3 

The components of 
demand and supply for 
user groups 

Ramirez (2000); 
adapted from 
Melody (1996) 

Definition of the dimensions of 
supply and differentiation between 
the demand by organizations vs. 
residential users 

Supply and demand 

4 

The multiple 
dimensions of ICTs 

Mansell and Wehn 
(1998) 

Locates services along four 
quadrants on the basis of four 
axial themes: transport, products, 
content and services 

Four axial themes: 
transport, products, 
content and services 

5 

The characteristics of 
the major periods of the 
Internet 

Bar et al. (2000) 

Characterizes the features of the 
Internet through its three major 
periods of evolution 

Evolution through time 

6 

Mapping indicators: a 
footprint analysis 

Mansell and Wehn 
(1998) 

Multiple indicators across country¬ 
wide sectoral statistics 

Food print notion: 
multiple indicators 

7 

Harvard’s E-readiness 
guide 

Information 

Technologies 

Group (n.d.) 

Multiple indicators grouped by 
infrastructure, network learning, 
network society, network economy 
and network policy 

Notion of readiness 
across five dimensions 

8 

The Australian 
technology map 

Houghton (1999) 

Sixteen quadrants locating 
standard industry indicators using 
similar axial themes to no. 4. 

Comprehensive means 
of tracking and showing 
relationships among 
indicators groups 

9 

A framework for rural 
and remote information 
and communication 
technologies 

Ramirez (2001b) 

Describes the interplay between 
policy, technology, community and 
organizational variables in shaping 
community-based networks 

Models the factors that 
shape a community 
network 

10 

Residential indicators 
of outputs 

Ramirez (2000) 

A review of common indicator 
groups from standard sources 
(emphasis on the Canadian 
context) 

Suggests the need for 
contextual, goal- 
oriented approaches to 
make sense of the 
myriad of indicators 


analysis of the third generation Internet, where the applications that will evolve are considered 
unpredictable. Urban planners acknowledge the difficulty in ascertaining ICT impact (Cohen, 
Salomon, & Nijkamp, 2002) and telecommunication planners acknowledge the limited 
predictability of impact (Bar et ah, 2000; Fink & Kenny, 2003; Mitchell, 2003). How people 
use technology depends on them having a taste of its potential, on having a safe and comfortable 
setting where they can play with it, and then their creativity emerges (Ramirez, 2001). Some 
authors see mistakes as ‘creative error’; man’s inability to predict its creativity suggests it is a 
necessary—though unreliable—ingredient for innovation (Sawhney, 2001). 
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Fig. 5. A framework for rural and remote information and communication technologies (Ramirez, 2000). 

We have now outlined four themes that require attention in agreeing how to measure impact: 

1. Community involvement in indicator selection 

2. Organizational dimensions 

3. Individual skills and capacities 

4. Limited predictability of the telecommunication services that emerge. 

The first item calls for participatory approaches in the planning of telecommunication 
infrastructure and services. The second and third items remind us that organizational and 
individual issues of acceptance, integration, skill, and familiarity are of strategic importance. The 
fourth item signals the fact that the path is only partially mapped, and that adjustments to 
strategy and measurement are unavoidable. 


5. Ways forward—a theoretical foundation 

Systems thinking and participatory action research are two sets of approaches that provide 
practical tools to respond to the context described above. In this section, they are briefly described 
as building blocks for a way forward. 

Systems thinking is holistic; it addresses overall patterns and relationships rather than reducing 
issues to smaller parts, which is the tendency of engineering approaches (Bennetts, Wood-Harper, 
& Mills, 2000). Systems thinking is useful as a tool for learning about complex situations and for 
interdisciplinary research (Ackoff, 1969). Since the late 1950s and early 1960s, systems thinking 
has also been applied to the analysis of organizations (Churchman, 1971; Emery & Trist, 1969; 
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Emery, 1969). What is relevant is the perspective that social and technological issues cannot be 
addressed in parallel; rather, they are to be analyzed jointly as a system and with the involvement 
of the stakeholders. 

In a variation of systems thinking, Checkland and Scholes (1990) developed soft systems 
methodology, SSM. One key contribution of SSM is the notion that the key stakeholders involved 
in a system are owners of the problem or issue. In other words, those who are directly affected, 
‘own’ the problem and should be involved in understanding and addressing it. SSM is therefore 
compatible with participatory action research approaches. 

SSM and participatory action-research have been applied to the analysis and conceptualization 
of complex systems, both in organizational management and in natural resource management. In 
the authors’ experience they are equally relevant to telecommunication services action-research. 

Soft systems methodology (SSM) originated from management science. It is an organized way 
to tackle real world ‘messy’ situations. A challenge arises when the problem is ill defined, 
complex, changing, messy and plagued with multiple interpretations (Checkland & Scholes, 
1990). SSM takes a systems approach to explore problem situations where there is no fixed 
outcome (Moores & Gregory, 2000) and where multiple stakeholders’ perspectives need to be 
accounted for. 

There are a growing number of examples of SSM being applied to telecommunication services 
research and planning (Kidd & Leung, 2000; Bennetts, et ah, 2000; Andrew & Petkov, 2000; 
Ramirez, 2001). SSM fits nicely into the ICT context where multiple parties are involved and have 
different perspectives, technical backgrounds, and terminologies; what communication writers 
refer to as different ‘languages’ (Peters, 1999; Jansen, 1989). In SSM, all of these stakeholders are 
seen as being ‘co-owners of the problem’, and are therefore in need of involvement in defining 
what it is that impact really means. 

SSM emanates from systems theory where particular elements become immediately relevant: (1) 
the appreciation of multiple interrelated parts, (2) the notion of hierarchies, and (3) the notion of 
emerging properties. The first one is self-explanatory, but does pose a challenge to conventional 
research on linear, causal relationships. The second sheds light on the fact that any system has 
different levels, or nested hierarchies, that merit attention. In the telecommunications services 
field, this means we need to pay attention to at least three levels: individuals, their organizations 
or sectoral interests, and the communities they live in. The third element suggests that oftentimes, 
the outcomes of the system are not predictable as they in fact emerge from patterns of interaction 
that were not possible before. As mentioned earlier, in the Third Generation Internet age, there is 
growing evidence that the types of services that will emerge are difficult to predict (Bar et ah, 
2000 ). 

In participatory-action research, making research both relevant and practical to the different 
groups and organizations involved in rural and remote telecommunication is a necessity. Action 
research is an established field in educational theory (Carr & Kemmis, 1983) and adult education 
(Kidd, 1973; Hall, Gillette, & Tandon, 1982). Adult education has been defined as a process that 
allows adults to become inner-directed, self-operated learners with an enlarged capacity for life 
and creativity (Kidd, 1973). The ‘research’ element in critical theory addresses the same goal: to 
transform the self-consciousness of individuals (Carr & Kemmis, 1983). Adult and popular 
education place emphasis on involvement by people in creating knowledge (Hall et al., 1982). 
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Creating knowledge is research—and people involvement is participation. Participation in 
appraising problems, in implementing projects and in evaluating their impact has become a 
central concern in community and rural development in the last three decades (Ramirez, 2000). 

It is clearly apparent that there is growing interest in approaching impact evaluation on the 
basis of an approach where the users are involved in defining what it is they seek to accomplish 
(Menou & Potvin, 2000; Stoll, Menou, Camacho, & Khellady, 2002). Systems thinking, and in 
particular SSM, and participatory action research, provide theoretical and methodological tools 
that have already been developed in other fields and that promise to contribute to building 
approaches to measuring the multiple impacts that telecommunication services may contribute to 
rural and remote communities. 


6. Experience to date: a case study 

On the basis of the above theoretical foundation and in direct response to the four themes 
described earlier (community involvement in indicator selection; organizational dimensions; 
individual skills and capacities; limited predictability of the telecommunication services that 
emerge), the authors have developed an approach for measuring the impact of telecommunica¬ 
tions services. It is worth emphasizing that the approach is still under construction: at the time of 
writing it is still at the implementation phase. The features of the approach are the following: 

1. Tracking indicators at three levels: community-wide measures of accessibility, sectoral 
indicators, and discerning relative changes in individuals’ skills and knowledge. 

2. Sectoral action plans. Engagement of relevant stakeholders from the start of a project to 
envision where the community wants to go in terms of improved well-being and services, and 
putting telecommunications equipment and services to work in response to those visions This 
involves gathering indicators of results and of intermediary outputs from the stakeholders on a 
sector by sector basis (namely, health, education, local government and economic 
development). 

3. Audio-visual testimonials: Using other qualitative tools, especially video to capture real-life 
stories about how people are harnessing the technology. 

The approach emphasizes that monitoring and evaluation efforts are best addressed during the 
planning of telecommunication services, when relevant stakeholders can be consulted and engaged 
in brainstorming what they wish to do with the technology. The methodology addresses three 
hierarchies within a community context: a community-wide dimension, a sectoral or activity- 
based dimension, and a personal dimension. The concept of nested hierarchies (from systems 
thinking) is of relevance here in that the community is at the lower end of the 
region-nation-global hierarchy, and the evaluation addresses three hierarchies within the 
community level. The reference to three levels of hierarchy reflects a systems orientation for 
monitoring and evaluation.^ 

^This approach is not unique, for example. Statistics Canada began exploring a systems-oriented approach to assess 
the impact of science and technology some years back (Statistics Canada, 1998a, b). 
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Experience has also shown that a temporal dimension deserves attention. In the short run, 
accessibility indicators become evident (e.g. number of computers in the community); in the mid¬ 
term, service delivery is measurable (e.g. number of students enrolled in the Internet high school; 
number of telemedicine sessions via videoconferencing) and people are able to voice opinions 
about the changes on video; lastly, impact in terms of community goals or visions take time, often 
beyond the duration of a project intervention. 

The context: K-Net Services, Northwestern Ontario, Canada. K-Net (K-Net.ca) is a regional 
information technology and content development organization that is working directly with five 
First Nations in the NAN of northern Ontario. K-Net supports and manages various local First 
Nation telecom initiatives across the region by delivering a variety of broadband services and 
developing electronic indigenous applications. Keewaytinook Okimakanak is a non-political 
Chiefs Council that advises and assists their member First Nations in the Sioux Fookout District 
of Northwestern Ontario. The Chiefs of the member First Nations that form the Board of 
Directors direct the organization. K-Net Services of Keewaytinook Okimakanak Northern 
Chiefs Council develops and maintains the network. Approximately 2800 people live in 
Keewaytinook Okimakanak communities. Deer Fake is the largest community with a total 
population of 850. There are 314 people living in North Spirit Fake, 316 people living in Poplar 
Hill, 470 people live in the community of Fort Severn and 539 people reside in Keewaywin. The 
territory where K-Net is present is about 200,000 km^ in size, with approximately 20,000 people 
from 23 First Nations north of Sioux Fookout (0.1 persons/km^). Most of the communities are 
only accessible by small plane, and during a few weeks by winter roads (Ramirez, 2000). 

The TeleCommons Development Group (TDG) is a consulting organization specializing in 
rural and remote telecommunications and both authors have been involved with this group. TDG 
has had extensive experience with research and planning interventions world wide, especially in 
the area of participatory planning and evaluation. The TDGand K-Net began applying the above 
approach in the five aboriginal communities directly supported by K-Net as of 1999-2000 
(Keewaytinook Okimakanak, 2000)."^ K-Net will be implementing the Kuh-ke-nah network of 
Smart First Nations (smart.K-Net.ca) from 2000 to early 2004, as a national demonstration 
project under Industry Canada’s Smart Communities Project.^ 

At the time of writing, the following actions have been implemented: 


• Development, testing and distribution of a household survey covering all homes in the five 
communities (2000, 2001, 2002; the 2003 data was under review at the time of writing). 

• Development, testing, distribution of an organizational and businesses survey in the five 
communities (2001, 2002; the 2003 data was under review at the time of writing). 

• Analysis of the survey data and feedback at meetings in all of the five communities (2000, 2001 
and 2002 data). 

• Community engagement workshops in each of the five communities to consult on sectoral 
visions and gather indicators (2001-2002; with initial workshops to develop the approach in 
1999). 

‘^Readers are referred to (Ramirez, 2000, 2001; Richardson & Ramirez, 1999) for further background. 

^For reference to the other Smart Demonstration Projects across Canada, and the overall programme, refer to http:// 
smartcommunities.ic.gc.ca. 
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• Video training and interviews in late 2003 and early 2004. 

• Preparation of a multi-media production with video clips and written case studies, 2004.® 
The approach addresses three levels: 


Level 


Tool 


Community-wide 
access to ICTs 
Sector-specific plans 


Individual skills and 
knowledge 


Annual survey of residences and organizations 

Community engagement workshops to define goals and objectives in 
health, education, local government and business and how ICTs can 
augment those goals, complemented with video testimonials by people 
working in community agencies that are offering new services such as 
tele-health and internet high school 

Video testimonials where people express their views on the role and 
impact of the technology 


Community-level access: This first dimension is addressed via a survey to track baseline 
indicators and provide a portrait of a community’s level of access to information and 
communication technology. This element is labelled the “community telecommunications 
portrait” and it serves mainly to provide a baseline portrait of access. The indicators are selected 
from standard telecommunications indicators in the literature, both Canadian and international 
(Fig. 6). 

The TDG and K-Net staff have already completed three rounds of house-to-house surveys and 
a fourth was implemented during the fall of 2003. By the end of the project 4 years of data would 
have been collected that will show the dramatic changes in communities that had no phone access 
before the end of the century and are now utilizing high-speed videoconferencing facilities.^ Fig. 7 
provides one example with cumulative data for one community over the first 3 years of the 
project. The most used Internet services are e-mail, the use for educational purposes and chat 
groups. 

6.1. Sectoral action plans 

The second element of the approach addresses sectoral plans developed during community 
engagement workshops (www.smart.K-Net.ca). Representatives from the major sectors of the 
community were invited for a 1-day planning session; these covered health, education and a 
combination of local government and economic development. The approach was labelled a 
results-based management framework. The focus is on how people want to use the technology and 
how they expect it will change their lives. While the design stems from logical framework analysis, 
in this case it is developed as part of a community engagement and planning process. This creates 

®http://smart.knet.ca/kuhkenah_flash.html. 

variation of the survey has been developed to document ICT access and use by businesses and agencies; it has been 
applied in other parts of Ontario and in Bolivia, South America. 






312 


R. Ramirez, D. Richardson / Telecommunications Policy 29 (2005) 297-319 


Indicator 

1 TU(1999) 

Indicator 

names 

HIFE, 1997 

Indicator 

names 

Census, 1996 

UNDP Human 
(used for the 
HD INDEX) 

Mansell & 
Wehn 
INDECES 

Cable 

I965C 

CABLE TV 




Cellular Phone 

I 271 

CELPHONE 


Mobile cellular 
telephone 
subscribers 
per 100 
people 


Fax machines 


Fa 


X machines 
per 100 
people 


Home 

Computer 

1 422 

COMP 


Personal 
computers per 
100 people 

Personal 
computers per 
capita 

Home 

Computer with 
modem 

14133 

Estimated 
modems in 
use 

COMPWMOD 




Access to 
make / receive 
emergency 
call 

EMERGC 

ALL 




Use of internet 

1 4221 

(equivalent) 

INTERNET 


Internet users 
per 100,000 
people 


Number of 
Colour TV sets 

1 965 

NCTVS 




Number of 
Telephone 
numbers 

NF 

ONNUMB 




Number of 
Telephones 

1 91 (main 
lines/100 
inhabitants 
"teledensity) 

NPHONE 

>98 

NTELPHON 
(Ag.Census 
1996) 

Main 

telephone 
lines per 100 
people 

Main 

Telephone 
lines per 
capita 

Number of 
radios in HH 

1955 

NRADIOS 


Radios per 

1,000 people 


Number of TV 
sets 

1965 

NTVS 


Televisions 
per 100 
people 

Number of TS 
sets per 100 
population 

Number of 
persons in 
household 
(HH) 

1 61 POP 

1 62 HHs 

PERINHH 

Ave. no. of 
persons in 
married or 
common-law 
families 



Number of 
households 
(HH) 



Number of 
private 
occupied 
dwellings 




Fig. 6. Residential indieators of outputs (Ramirez, 2000). 


a context in which stakeholders are invited to identify relevant indicators. Its major contribution 
is the definition of a logical sequence of activities or outputs, outcomes and results. 

The tools that are used come from participatory action research work and involve hands-on 
activities to visualize complex situations and focus on visions of a desirable future (Lightfoot et ah, 
2001). Participants were invited to define where they wanted to go with their services, what 
programmes are in place to achieve those goals, what telecommunications technologies and services 
are needed to enhance the programmes, and what indicators should be used to track impact. 

For example, in one community the health sector team developed a list of results and linked 
these to the programmes that could help achieve the results. They provided a number of indicators 
to verify that the results had been achieved as well as others to verify that the programmes had 
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Purpose of Internet use 
n (2000) = 124 (2001) = 309 (2002)= 441 



□2000 02001 02002 


Fig. 7. The most used Internet services in Keewaywin First Nation, Ontario, between 2000 and 2002 (TeleCommons 
Development Group, 2002). 


been implemented. The telecommunication technologies that could enhance those programmes 
were included between brackets. 


Results 

Indicators of results 

Programmes 

Indicators of programme 
impact 

High esteem 

Less violence, suicide 

Social activities 

More community 


rate drops, less 

(computer club), mental 

involvement in various 


vandalism; people 

health fitness 

programmes, i.e. fitness 


involved in more positive 
activities, people 
working to keep their 
land clean; sense of pride 

programme 

programmes, guitar 
lessons 

Stress 

Less violence, better 

Guitar lessons, therapy. 

More community 

management 

overall health, better 

organization and 

involvement in various 


quality of work 

coordination (list serve), 
fitness programme, 
leisure and recreation 

programmes, i.e. fitness 
programmes, guitar 
lessons 

Healthy 

Better nutrition, more 

Elder support (video 


foods 

traditional foods, class 
room visits, more 
healthy foods in store 

conferencing), healthy 
babies, homecare, 
nutrition 
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Health Sector 

Band administration 
(locai government 
office 

Education Sector 

Economic deveiopment 

More people involved in the diabetic 
programs 

No deficit 

First nations businesses with 
outside partners 

Parents and community talking 
to children in native language 

More moms that are pregnant are 
aware of the effects of drugs, 
smoke, and alcohol abuse. 

Improved staff relations 

Expanded services and 
opportunities 

Number of WAHSA students 
who have completed the 
program 

Better nutrition 

Number of video 
conferences 

More jobs 

More elders accepting 
technology 

More traditional foods 


New businesses 

Adults enrolled in school 

More community involvement in 
various programs, i.e. fitness 
programs, guitar lessons 

Monthly budget 
meetings 

Reduced welfare 
dependency 

Getting computers for special 
needs 

Better overall health 


Better community 
environment 

Individual Web sites 

Better quality of work 



School Web site 

Fitness program 


Business plans 

Proposals 


Social activities 


Better maintenance for 
existing road 

Using syllables keyboards 

Home care 


Veggie garden 

Training sessions at E-centre 

Hygiene awareness 


Better water and sewer 

More people using on-line 
training 

Housing awareness 


Better houses 

Number of students in grade 4 
-8 

Better maintained homes 


More equipment that works 

Number of career days 

Dentist and eye doctor 




Referral clerk 




Workshops and training 




More patients treated in community 




Better awareness of diabetes 





Fig. 8. Summary of indicators of results provided by the community of Keewaywin, Northwestern Ontario, Canada 
(TeleCommons Development Group, 2002). 


It is worthy underlining that this process is not comprehensive, as the participants did not have 
experience with logical frameworks and planning. However, the process constitutes an important 
capacity building event to help people plan their development while harnessing the information 
and communication technologies. These participatory planning events are often not integrated 
into telecommunication planning in remote communities. 

Fig. 8 provides a sample of the types of indicators that the community members contributed. 
The telecommunication services are to contribute towards the accomplishment of these goals, and 
the indicators reflect how the community describes the benefits. 

The third element refers to people’s skills and knowledge changes, something that is best 
captured in their own terms. The authors provided tools based on methods of adult education 
where people design their own learning objectives (Knowles, 1975). The tool was tested in the 
context of an evaluation of a rural library project in southern Canada. However, in the K-Net 
demonstration project, these tools have had limited acceptance. However, the authors have come 
to realize that people harness technology and make it work for them in unpredictable ways. The 
surveys will never capture that richness. Real life stories of how telecommunication services can 
bring about new opportunities and perspectives are best captured with audio-visual media. Video 
technology lends itself to involving young people in using the tools in a creative manner. 
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Testimonial video-clips capture personal perspectives about how the technology is changing 
livelihoods today. They introduce a tool to gauge current personal and organizational 
experiences. Video-clips and oral histories are already a part of the K-Net.ca website (see: 
www.legends.K-Net.on.ca) and are now also part of a multi-media production that includes five 
case studies (Ramirez, Aitkin, Jamieson, & Richardson, 2004). The importance of narrative as an 
evaluation tool becomes particularly relevant in a culture that has a strong oral tradition. Digital 
video has the added advantage of the ease of computer-based editing so that in the very near 
future, productions can be done by community members themselves (Ferreira, Ramirez, and 
Walmark (2004); Ferreira, Walmark, Kenny, and Ramirez (2004a)). 


7. Conclusion 

This article describes an approach for measuring impact and makes reference to some practical 
community engagement and monitoring work that is in progress. This work will yield time-series 
data that has thus far not been available in Canada. A number of elements require further 
attention. If a telecommunication investment demonstrates that a service can be delivered for a 
fraction of a price, one can only wonder how long it will take until the evidence is used to justify a 
cut in funding. 

It has been demonstrated that the cost of delivering this [telemedicine] service as an ongoing 
program, with 2-3 clients drawn from all of the Keewaytinook Oldmakanak First Nations 
Communities being seen during weekly 1.5-hour teleconferencing sessions, averaging 4 per 
month over 12 months, is estimated to be as much as $985 per client-session (and potentially, 
even less). This estimate is significantly less than the overall cost of delivering this service by 
flying clients out to the regional First Nations counseling centre in Sioux Lookout, estimated to 
be $2,716 per client-session in this study (Centre for Health Services and Policy Research, 2002: 
p. 49). 

It is important to mention that in many cases the savings are being made by the external 
funding agencies. The extent to which these savings will be translated into improved and 
continued support to the communities remains to be seen. 

Another limiting factor is the cost of tracking indicators: who is able and willing to spend the 
funds required for what is essentially a monitoring and evaluation activity? Who actually stands 
to gain from tracking indicators? In Canada, governmental incentive programmes are increasingly 
emphasizing the impact assessment dimension; and this becomes a requirement for project 
implementers and grant providers. However, the procedures for measuring impact assessment 
thus far remain undefined, which in itself may prove to be an opportunity for creativity and 
innovation. 

This measurement approach is associated with a demonstration project of national stature, 
hence it has significant policy implications. The context of the remote aboriginal communities is 
drastically different from that of rural Canada; what emerges from this project will shape future 
programmes for the north. The low population density of remote communities means that there is 
at best a weak business case for the infrastructure in terms of revenue from the user base. 
However, there may be substantial savings for the agencies that provide the services. At the 
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community level, there are substantial human and social benefits that can translate into savings in 
other remedial programmes: keeping children in school longer may mean young people with 
higher self-esteem and fewer cases of teenage drug abuse. The recent case studies and video 
reports on the progress achieved with the K-Net demonstration project point towards the need to 
address measurement through broader lenses, including the sustainable livelihoods framework.* 
The approach described in this paper is new in terms of how it combines quantitative tools, 
participatory planning using conventional tools, and video testimonials. The authors feel that this 
combination of methods has the potential to capture the impact at its different levels, and through 
the multiple perspectives of the stakeholders involved. 
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